@MitchBradley : FYI.

I updated the CI workflow to compile against your AsyncTCP posix version instead of the one from our repo using lwip and FreeRTOS functions.

I can produce an executable, which, when launched, will listen to port 8080.

But a curl -v http://127.0.0.1:8080/ leads to nothing FYI.

Great, now that this piece is stable, I can proceed to the next step. I will figure out why curl does nothing with your test case, then work on a focused test case for the close state machine problem. In my full FluidNC build, everything seems to work - serving web pages, interacting via a REST API, websockets, and WebDAV, and simultaneously simulating a CNC machine in a web app that is connecting back on a websocket.

The problem with the example is that it needs

  #include <AsyncTCP.h>
  ...
  setup() {
    PosixAsyncTCPManager::getInstance().begin();

There is probably a "more compatible" way; Arduino-Emulator has an Ethernet class and a WiFi class mock on top of it. I need to study the other AsyncTCP implementations to see how they are initialized.

The problem with the example is that it needs

  #include <AsyncTCP.h>
  ...
  setup() {
    PosixAsyncTCPManager::getInstance().begin();

There is probably a "more compatible" way; Arduino-Emulator has an Ethernet class and a WiFi class mock on top of it. I need to study the other AsyncTCP implementations to see how they are initialized.

I confirm, it works perfectly! I will push the fix, thanks!

Actually, I think I can do better. I think that the call should be embedded in PosixAsyncTCP > AsyncServer.begin().

Actually, I think I can do better. I think that the call should be embedded in PosixAsyncTCP > AsyncServer.begin().

👍

Quick question also regarding AsyncTCP... I tried to compile the AsyncTCP we have (based on lwip). The only issue I had was all the FreeRTOS usage we do in AsyncTCP, but I was able to git clone and compile lwip repo.

Do you think that if we were using std::thread and another queue system (and semaphore) the library could be reused as-is for the emulator ?

Because if yes, then maybe we could try to refactor AsyncTCP to support the HOST macro ?

What was the driven factor for the reimplementation of AsyncTCP ?

PosixAsyncTCP now starts the polling thread automatically in AsyncServer::begin(), so the explicit call in setup() is unnecessary, albeit harmless.

Re the reimplementation - it seemed like the easiest path given the fact that every host system implements ultra-stable BSD networking out of the box. lwip requires a lot of configuration and I imagined spending inordinate amounts of time fighting it - and it would still be sitting on top of the host's BSD networking, so an extra layer of complexity, configuration, and dependency management for no benefit. Copilot, backended by ChatGPT and Claude, did most of the heavy lifting for the implementation.

I have a FreeRTOS mock that I use in FluidNC, implementing queue, semphr, task, and timers on top of C++ mutexes, semaphores, threads, and chrono. I only implemented what my app needs. Right now it is embedded in the FluidNC code base, not a published package.

One FreeRTOS feature that I probably cannot replace is queues, since I need the ISR-safe versions and C++ doesn't have that. Maybe I could use the Embedded Template Library but that would be yet another framework to learn to use.

That’s an impressive work!

And did you considered using AsyncTCPSock ? This one is bsd socket based also.

I did not know of the existence of AsyncTCPSock. Had I known, I probably would have used it. I am looking at it now.

Ah, I see what is happening. AsyncTCPSock sits on top of lwip, and depends on FreeRTOS and has some ifdef-guarded ESP dependencies. I think its primary difference from the other AsyncTCP is its use of lwip's BSD socket layer instead of the netconn interface or whatever that is called. So my concerns about adding an unnecessary lwip layer on top of an existing TCP hold for it too.

Thanks for pointing it out.

Hi @MitchBradley ,

After updating to Arduino 3.3.9 (latest), the arduino emulator example is not compiling anymore because they now include #include <freertos/FreeRTOS.h> directly in FSImpl.h

You can see this build error: https://github.com/ESP32Async/ESPAsyncWebServer/actions/runs/27000486972/job/79679597605#step:6:64

In file included from /home/runner/work/ESPAsyncWebServer/ESPAsyncWebServer/.ci/arduino-esp32/libraries/FS/src/FS.cpp:22:
/home/runner/work/ESPAsyncWebServer/ESPAsyncWebServer/.ci/arduino-esp32/libraries/FS/src/FSImpl.h:25:10: fatal error: freertos/FreeRTOS.h: No such file or directory
   25 | #include <freertos/FreeRTOS.h>
      |          ^~~~~~~~~~~~~~~~~~~~~

Hi @MitchBradley ,

After updating to Arduino 3.3.9 (latest), the arduino emulator example is not compiling anymore because they now include #include <freertos/FreeRTOS.h> directly in FSImpl.h

You can see this build error: https://github.com/ESP32Async/ESPAsyncWebServer/actions/runs/27000486972/job/79679597605#step:6:64

In file included from /home/runner/work/ESPAsyncWebServer/ESPAsyncWebServer/.ci/arduino-esp32/libraries/FS/src/FS.cpp:22:
/home/runner/work/ESPAsyncWebServer/ESPAsyncWebServer/.ci/arduino-esp32/libraries/FS/src/FSImpl.h:25:10: fatal error: freertos/FreeRTOS.h: No such file or directory
   25 | #include <freertos/FreeRTOS.h>
      |          ^~~~~~~~~~~~~~~~~~~~~

@MitchBradley For now I have fixed the CI to v3.3.8: a87fcdb

Note: I was not able to open an issue in your repo since the issue tab is not activated.

Okay I will take a look at it over the weekend.

Okay, so here is my take on this.

FS is a virtualization layer on top of SD, SPIFFS, etc. It exists in ESP Arduino and in RP2040 Arduino (though their APIs are not entirely the same), but is not in Arduino-Core. Arduino-Emulator does not have it (yet).

The ESP FS code had some race conditions that were fixed in 3.3.9 by introducing locks that resolve to FreeRTOS semaphores.The lock class "FSLockGuard" is defined in FSImpl.h, introducing the FreeRTOS dependency, but the uses of FSLockGuard are in vfs_api.cpp, making me wonder if FSImpl.h is the wrong place to introduce the dependency. I do not know whether the rp2040 FS suffers from the same races (maybe not, if rp2040 file operations are thread-safe at a lower level).

ESPAsyncWebServer depends on FS. using it to serve static files. My FluidNC app serves static file using a different method that involves a C++ std::filesystem implementation. When using Arduino-Emulator, I satisfy the FS dependency with mocking code that is essentially the old FS.h, FS.cpp, and FSImpl.h, but I never mount anything.

I see two ways forward:

1. In examples/arduino_emulator, include the old, FreeRTOS independent files FS.h, FS.cpp, and FSImpl.h. That main.cpp does not have any routes to serve static files, so that probably would satisfy the compiler. The ESP arduino framework would thus not be needed.
2. Add the FS class to Arduino-Emulator. That would be the best long-term approach is an ideal world, but it would require coordination, testing and maintenance that might not pay off.

Thoughts?

Thank you for the analysis. Option 1) could be done for now until option 2 is available what do you think ?

@me-no-dev FYI

I said that FS is virtualization on top of SD etc. That is not quite correct; it is actually a base class for SD etc. But the rest of the comment is essentially correct.

FS first appeared for ESP8266. The pico implementation is almost identical to the ESP8266 implementation, while the ESP32 version has diverged a lot, based on the use of ESP-IDF's vfs layer. ArduinoCore-API does not include it. It think that was intentional. Due to the API differences and the lack of support on some Arduino cores, it is not very portable.

FS layer in ESP32 is actually based on the original Arduino SD lib. Trying to expose the same API for all file systems. ESP8266 was more based on SPIFFS

Could you check if espressif/arduino-esp32#12670 solves this issue ?